Plugins created in DEVONagent Pro's Plugins & Scanners window, are saved in ~/Library/Application Support/DEVONagent/Plugins. The format of the plugin depends on whether you choose to edit the plugin in Plist (.xml) or JSON (.json) format. Optionally, you can work in an external application. XML-based plugins can be created or edited in applications like BBEdit or even Apple's Xcode. JSON-based plugins are plain text and can be created and edited in any plain text editor, for example TextEdit. Externally created plugins should be saved with the proper file extension in the same location mentioned above.
When you create a plugin in the application, certain keys will be automatically added for you. Double-click the plugin in the Plugins & Scanners window to view and edit the keys. This is a good way to learn the internal structure and workings of a DEVONagent Pro plugin. It's easy to make changes and test them quickly using this window. However, you can use in an external editor, if you choose.
Listed below are the available keys, some required and some optional. Also, if the source to be processed is not HTML but XML or JSON output, there are specific keys that need to be used. Those are listed separately.
Boolean values are true or false and string values are case-insensitive.
Keys for HTML
Required Keys:
Name: [string or dict] The name of the plugin. Example: BBC News.
EngineUrl: [string or array of strings] The Internet address(es) of the search engine. The strings _agentQuery_, _agentNumber_ and _agentOffset_ are replaced by the search term, the number of results and the offset of the current page. _agentNumber_ and _agentOffset_ are optional. _agentLanguage_ is replaced by the language used by the plugin (ISO 639 two-letter codes). Can contain the placeholders _agentUser_ and _agent_Password_ for HTTP authentication.
Optional Keys:
CrawlDelay: [float] Specify a delay between two requests sent to the same server, defined in seconds.
DateEnd: [string or array of strings] String marking the end of the range of the HTML source code containing the date of a page.
DateStart: [string or array of strings] String marking the beginning of the range of the HTML source code containing the date of a page. Note that DateEnd is optional if DateStart is specified. Useful if server provides no dates or the wrong dates.
EncodingPage: [string] Default character encoding for result pages (default: Latin-1). Examples: ISO-8859-1, UTF-8, ISO-8859-2, Shift-JIS.
EncodingUrl: [string] Character encoding for search term _agentQuery_ (default: Latin-1). Examples: ISO-8859-1, UTF-8, ISO-8859-2, Shift-JIS.
EngineNextUrl: [string or array of strings] The Internet address(es) of the search engine(s) used for subsequent pages (if not identical to the address of the first page). The strings _agentQuery_, _agentNumber_ and _agentOffset_ are replaced by the search term, the number of results and the offset of the current page. _agentNumber_ is optional.
EngineRelated: [string] Defines a URL for a 'see also' function of the search engine.
EngineSite: [string or array of strings] Defines (a) (sub)domain(s) with or without a path to search. DEVONagent Pro uses an appropriate common search engine to perform the search. Example: devontechnologies.com/products/.
ExcludeLinksNamed: [array of strings] Defines an array of names of links which should be skipped.
FileTypes: [string] List of all supported file types: html, feed/rss, text, pdf, OpenOffice, Microsoft/Word, RTF or all. Default is all.
FilterLanguage: [Boolean] Enable or disable filtering by language.
FilterSimilarPages: [Boolean] Enable or disable DEVONagent Pro's Similar Pages filter (default: true).
GoogleTags: [Boolean] Defines whether googleon/googleoff tags should be supported.
HTTPShouldHandleCookies: [Boolean] Defines whether cookies should be handled by HTTP or not.
Keyword: [string or array] One or more unique keywords used for quick searching (like "Mozilla Keywords") by entering the keyword followed by the search term in the URL field of the browser window.
KeywordsUrl: [string] Defines an alternate URL used for the Mozilla-like keywords in the browser's address bar.
LinksEnd: [string or array of strings] String marking the end of the range of the HTML source code containing the links to use (any position after the last link to use but before the next link). Example: -- Footer Area --.
LinksMatching: [string or array] Only links of result pages matching this string or these strings are used (prefix/suffix * wildcards are supported). Examples: *news.bbc.co.uk/*.
LinksNotMatching: [string or array] Only links of result pages not matching this string or these strings are used (prefix/suffix * wildcards are supported).
LinksStart: [string or array of strings] String marking the beginning of the range of the HTML source code containing the links to use (any position before the first link to use and after the last link to ignore). Note that LinksEnd is optional if LinksStart is specified. Example: search results for.
LoginPost: [string] Defines the optional HTTP POST parameters. Usage like Post. Can contain the placeholders _agentUser_ and _agent_Password_.
LoginURL: [string] The address used to log into a website. Usage similar to "EngineUrl". Can contain the placeholders _agentUser_ and _agent_Password_.
NextLinkStart: [string] Defines the start of the range for the 'next results page' link.
NextLinkEnd: [string] Defines the end of the range for the 'next results page' link.
NextLinkName: [string] Defines the name of the link for the 'next results page' (instead of having to use NextLinkStart/NextLinkEnd).
NoTopics: [array or string] Array of strings or string containing separated words to ignore in the topics list. Add site specific words here.
OffsetPerPage: [integer] Increase for _agentOffset_ after every page (default: 1).
Operators: [integer] Bitmap describing the possibilities of the search engine and the required syntax (see below). The default value is 59 (1+2+8+16+32), meaning that AND, OR, NOT, phrases, and parentheses are supported.
OperatorsDictionary: [dict] A dictionary containing key-string pairs to convert the name of operators to a different name required by the search engine. Possible keys are AND, ANDNOT, OR, NOT, XOR, NEAR, BEFORE, AFTER, -.
ParseLinks: [Boolean] Parse and unify links of common search engines (Google, Yahoo, Bing) (default: false).
Post: [dict or string] Send a HTTP POST instead of a HTTP GET action using this form data. Note: The strings of the form data can contain _agentQuery_ and _agentNumber_ placeholders.
PostScan: [Boolean] Tell DEVONagent Pro to check if the retrieved pages match the search term (true), or to accept all (false) pages (default: true).
ResultsPerPage: [integer] Number of results per page and maximal value for _agentNumber_ (default: 10).
RequiresCrawling: [Boolean] Marks this plugin as needing crawling.
RequiresThumbnails: [Boolean] Marks this plugin as using only thumbnailed links. Crawling needs to be skipped.
RoundResultsPerPage: [integer] Always round number of results per page (_agentNumber_) to multiples of this value (default: 10).
ScanTerm: [string] String specifying a term used to filter results (e.g., instead of or in addition to the secondary query defined by the user).
SkipsCrawling: [Boolean] Marks this plugin as skipping crawling.
Start: [integer] The initial value for _agentOffset_ (default: 0).
StripTags: [array] Array of strings marking HTML tags defining blocks to strip.
TextEnd: [string or array of strings] String marking the end of the range of the HTML source code containing the relevant text of a page. TextStart is optional if TextEnd is specified.
TextStart: [string or array of strings] String marking the beginning of the range of the HTML source code containing the relevant text of a page. Note that TextEnd is optional if TextStart is specified. Useful if you're querying a specific site (all results have the same layout) to skip headers, footers or navigation of a page. This improves plain text display, digest and summarization.
TextTags: [array] Array of strings marking HTML tags defining blocks to convert to text.
Timeout: [integer] Sets the timeout for requests; default is 60 seconds.
TitleEnd: [string or array of strings] String marking the end of the range of the HTML source code containing the title of a page.
TitleStart: [string or array of strings] String marking the beginning of the range of the HTML source code containing the title of a page. Note that TitleEnd is optional if TitleStart is specified. Only useful if all results have identical or no titles (for example a Usenet query).
UseListingResults: [Boolean] Treat the returned pages as results (and not as pages linking to the results). All Links... keys are unused in that case.
UserAgent: [string] Custom HTTP user agent string.
Keys for XML/JSON
Required Keys:
ResultsKeyPath: [string] The path to the results in the search engine's output.
LinksKeyPath: [string] The path to the links relative to ResultsKeyPath.
Optional Keys:
ContentsKeyPath: [string] The path to the contents relative to ResultsKeyPath.
DescriptionsKeyPath: [string] The path to the descriptions relative to ResultsKeyPath.
TitlesKeyPath: [string] The path to the titles relative to ResultsKeyPath.
ThumbnailsKeyPath: [string] The path to the thumbnails relative to ResultsKeyPath.
JSONPost: [Dictionary] Sends the content of the dictionary as HTTP POST request. The dictionary can contain nested dictionaries or arrays. string-placeholders, e.g., _agentNumber_ or _agentQuery_, are still replaced.
Localization
Three identifier keys of a plugin can be localized: Name, Info and Description. Specify the desired key as a dict (dictionary) containing key and string pairs, one per additional language. The key contains the appropriate language code, e.g., en, de, or fr, and the string is the localized text for the specified language.
Additionally, the following keys can be localized by adding a suffix for use with language-specific search engines: EngineUrl, EngineNextUrl, KeywordsUrl, and NextLinkName. Language identifiers are: DE, DA, NL, ES, HU, FR, IT, CS, PT, PL, SV, FI, EN, NO, RU, EL, AR, and HE.
Notes
Operator Bitmap: The following table gives you the necessary values for calculating the operator bitmap for the "Operators" key (see above):
0 [value: 1]: Supports AND operator
1 [value: 2]: Supports OR operator
2 [value: 4]: Supports XOR operator
3 [value: 8]: Supports NOT operator
4 [value: 16]: Supports "" phrases
5 [value: 32]: Supports () parenthesis
6 [value: 64]: Supports NEAR operator
7 [value: 128]: Supports AFTER operator
8 [value: 256]: Supports BEFORE operator
9 [value: 512]: Supports NEXT operator
11 [value: 2048]: Use ANDNOT instead of NOT or -
12 [value: 4096]: Convert search term to lower case
13 [value: 8192]: Convert search term to upper case
14 [value: 16384]: Engine scans for any word by default (not for all words) without an explicit operator
15 [value: 32768]: Use AND/NOT instead of +/-
16 [value: 65536]: Use | instead of OR
17 [value: 131072]: Supports OPT/ operator
LinksStart, et al.:LinksStart, LinksEnd, LinksMatching, and LinksNotMatching are three different ways to specify interesting links of a page. Specify none, some, or all of them depending on the search engine you are accessing. The more you specify, the higher the chances are that the plugin will still work after the structure of the HTML code or the links changed. First DEVONagent Pro retrieves links using LinksStart/LinksEnd, then it filters them using LinksNotMatching, and finally it matches them against LinksMatching.
Link Listings vs Matched Pages:LinksStart/LinksEnd, LinksMatching, and LinksNotMatching are used on pages listing links, e.g., a page of search engine results. Links are always followed on these pages. TextStart/TextEnd, TitleStart/TitleEnd, and DateStart/DateEnd are used on individual web pages, i.e., a matched Wikipedia page. The FollowLinks key/option can be used on these pages.
